Shouldn't you annotate the inverse relation? I.e. private Invoice parentInvoice? Not sure whether even JoinColumn is allowed here.

I tried that as well:

@ManyToOne(fetch = FetchType.LAZY)
@JoinFormula(value = "SUBSTR(hierarchy_path, 0, INSTR(hierarchy_path, '/') - 1)", referencedColumnName = "id")
private Invoice rootInvoice;

@OneToMany(fetch = FetchType.LAZY, mappedBy = "rootInvoice")
private Set<Invoice> childInvoices;

but there is same exception as described here https://discourse.hibernate.org/t/how-to-use-the-hibernate-joinformula-annotation-with-a-onetomany-jpa-association/1461 (i.e. ManyToOne works with JoinFormula, but OneToMany doesn't - even when "mappedBy" is added and JoinFormula is on ManyToOne side).
I guess the question is just whether JoinFormula works with OneToMany - but if it doesn't, how is anyone supposed to know that?

There is an existing test with a @JoinFormula @ManyToOne relation here: https://github.com/hibernate/hibernate-orm/blob/e5dc635a52362f69b69acb8d5b166b69b165dbbd/hibernate-core/src/test/java/org/hibernate/test/annotations/manytoonewithformula/ManyToOneWithFormulaTest.java . So what you're saying is that when this example is extended with an inverse mapping (a @OneToMany(mappedBy = ""), Hibernate fails to boot?

Yes, that's exactly what I was trying to say. For instance, in the test that you mentioned, if you add following to Language entity:

private List<Message> messages;

@OneToMany(fetch = FetchType.LAZY, mappedBy = "language")
public List<Message> getMessages() {
    return messages;
}

public void setMessages(List<Message> messages) {
    this.messages = messages;
}

the test fails with exception
Caused by: java.lang.ClassCastException: org.hibernate.mapping.Formula cannot be cast to org.hibernate.mapping.Column